Lecture 12: Recent Advances in Time Integration Methods and How They Can Enable Exascale Simulations

To prepare for exascale systems, scientific simulations are growing in physical realism and thus complexity. This increase often results in additional and changing time scales. Time integration methods are critical to efficient solution of these multiphysics systems. Yet, many large-scale applications have not fully embraced modern time integration methods nor efficient software implementations. Hence, achieving temporal accuracy with new and complex simulations has proved challenging. We will overview recent advances in time integration methods, including additive IMEX methods, multirate methods, and parallel-in-time approaches, expected to help realize the potential of exascale systems on multiphysics simulations. Efficient execution of these methods relies, in turn, on efficient algebraic solvers, and we will discuss the relationships between integrators and solvers. In addition, an effective time integration approach is not complete without efficient software, and we will discuss effective software design approaches for time integrators and their uses in application codes. Lastly, examples demonstrating some of these new methods and their implementations will be presented. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. LLNL-ABS- 819501

Nitrous oxide and methane in the Atlantic Ocean between 50 degrees North and 52 degrees South: Latitudinal distribution and sea-to-air flux

We discuss nitrous oxide (N2O) and methane (CH4) distributions in 49 vertical profiles covering the upper 300 m of the water column along two 13,500 km transects between 50°N and 52°S during the Atlantic Meridional Transect (AMT) programme (AMT cruises 12 and 13). Vertical N2O profiles were amenable to analysis on the basis of common features coincident with Longhurst provinces. In contrast, CH4 showed no such pattern. The most striking feature of the latitudinal depth distributions was a well-defined “plume” of exceptionally high N2O concentrations coincident with very low levels of CH4, located between 23.5°N and 23.5°S; this feature reflects the upwelling of deep waters containing N2O derived from nitrification, as identified by an analysis of N2O, apparent oxygen utilization (AOU) and NO3-, and presumably depleted in CH4 by bacterial oxidation. Sea-to-air emissions fluxes for a region equivalent to 42% of the Atlantic Ocean surface area were in the range 0.40–0.68 Tg N2O yr-1 and 0.81–1.43 Tg CH4 yr-1. Based on contemporary estimates of the global ocean source strengths of atmospheric N2O and CH4, the Atlantic Ocean could account for 6–15% and 4–13%, respectively, of these source totals. Given that the Atlantic Ocean accounts for around 20% of the global ocean surface, on unit area basis it appears that the Atlantic may be a slightly weaker source of atmospheric N2O than other ocean regions but it could make a somewhat larger contribution to marine-derived atmospheric CH4 than previously thought

Biological and physical forcing of carbonate chemistry in an upwelling filament off northwest Africa: Results from a Lagrangian study

The Mauritanian upwelling system is one of the most biologically productive regions of the world's oceans. Coastal upwelling transfers nutrients to the sun-lit surface ocean, thereby stimulating phytoplankton growth. Upwelling of deep waters also supplies dissolved inorganic carbon (DIC), high levels of which lead to low calcium carbonate saturation states in surface waters, with potentially adverse effects on marine calcifiers. In this study an upwelled filament off the coast of northwest Africa was followed using drifting buoys and sulphur hexafluoride to determine how the carbonate chemistry changed over time as a result of biological, physical and chemical processes. The initial pHtot in the mixed layer of the upwelled plume was 7.94 and the saturation states of calcite and aragonite were 3.4 and 2.2, respectively. As the plume moved offshore over a period of 9 days, biological uptake of DIC (37 ?mol kg?1) reduced pCO2 concentrations from 540 to 410 ?atm, thereby increasing pHtot to 8.05 and calcite and aragonite saturation states to 4.0 and 2.7 respectively. The increase (25 ?mol kg?1) in total alkalinity over the 9 day study period can be accounted for solely by the combined effects of nitrate uptake and processes that alter salinity (i.e., evaporation and mixing with other water masses). We found no evidence of significant alkalinity accumulation as a result of exudation of organic bases by primary producers. The ongoing expansion of oxygen minimum zones through global warming will likely further reduce the CaCO3 saturation of upwelled waters, amplifying any adverse consequences of ocean acidification on the ecosystem of the Mauritanian upwelling system

Enabling GPU Accelerated Computing in the SUNDIALS Time Integration Library

As part of the Exascale Computing Project (ECP), a recent focus of development efforts for the SUite of Nonlinear and DIfferential/ALgebraic equation Solvers (SUNDIALS) has been to enable GPU-accelerated time integration in scientific applications at extreme scales. This effort has resulted in several new GPU-enabled implementations of core SUNDIALS data structures, support for programming paradigms which are aware of the heterogeneous architectures, and the introduction of utilities to provide new points of flexibility. In this paper, we discuss our considerations, both internal and external, when designing these new features and present the features themselves. We also present performance results for several of the features on the Summit supercomputer and early access hardware for the Frontier supercomputer, which demonstrate negligible performance overhead resulting from the additional infrastructure and significant speedups when using both NVIDIA and AMD GPUs

Performance of explicit and IMEX MRI multirate methods on complex reactive flow problems within modern parallel adaptive structured grid frameworks

Large-scale multiphysics simulations are computationally challenging due to the coupling of multiple processes with widely disparate time scales. The advent of exascale computing systems exacerbates these challenges, since these enable ever increasing size and complexity. Recently, there has been renewed interest in developing multirate methods as a means to handle the large range of time scales, as these methods may afford greater accuracy and efficiency than more traditional approaches of using IMEX and low-order operator splitting schemes. However, there have been few performance studies that compare different classes of multirate integrators on complex application problems. We study the performance of several newly developed multirate infinitesimal (MRI) methods, implemented in the SUNDIALS solver package, on two reacting flow model problems built on structured mesh frameworks. The first model revisits the work of Emmet et al. (2014) on a compressible reacting flow problem with complex chemistry that is implemented using BoxLib but where we now include comparisons between a new explicit MRI scheme with the multirate spectral deferred correction (SDC) methods in the original paper. The second problem uses the same complex chemistry as the first problem, combined with a simplified flow model, but run at a large spatial scale where explicit methods become infeasible due to stability constraints. Two recently developed implicit-explicit MRI multirate methods are tested. These methods rely on advanced features of the AMReX framework on which the model is built, such as multilevel grids and multilevel preconditioners. The results from these two problems show that MRI multirate methods can offer significant performance benefits on complex multiphysics application problems and that these methods may be combined with advanced spatial discretization to compound the advantages of both

Implicit solvers for large-scale nonlinear problems

Abstract. Computational scientists are grappling with increasingly complex, multi-rate applications that couple such physical phenomena as fluid dynamics, electromagnetics, radiation transport, chemical and nuclear reactions, and wave and material propagation in inhomogeneous media. Parallel computers with large storage capacities are paving the way for high-resolution simulations of coupled problems; however, hardware improvements alone will not prove enough to enable simulations based on brute-force algorithmic approaches. To accurately capture nonlinear couplings between dynamically relevant phenomena, often while stepping over rapid adjustments to quasi-equilibria, simulation scientists are increasingly turning to implicit formulations that require a discrete nonlinear system to be solved for each time step or steady state solution. Recent advances in iterative methods have made fully implicit formulations a viable option for solution of these large-scale problems. In this paper, we overview one of the most effective iterative methods, Newton-Krylov, for nonlinear systems and point to software packages with its implementation. We illustrate the method with an example from magnetically confined plasma fusion and briefly survey other areas in which implicit methods have bestowed important advantages, such as allowing high-order temporal integration and providing a pathway to sensitivity analyses and optimization. Lastly, we overview algorithm extensions under development motivated by current SciDAC applications

Numerical coupling of aerosol emissions, dry removal, and turbulent mixing in the E3SM Atmosphere Model version 1 (EAMv1), part I: dust budget analyses and the impacts of a revised coupling scheme

An earlier study evaluating the dust life cycle in EAMv1 has revealed that the simulated global mean dust lifetime is substantially shorter when higher vertical resolution is used, primarily due to significant strengthening of dust dry removal in source regions. This paper demonstrates that the sequential splitting of aerosol emissions, dry removal, and turbulent mixing in the model's time integration loop, especially the calculation of dry removal after surface emissions and before turbulent mixing, is the primary reason for the vertical resolution sensitivity reported in that earlier study. Based on this reasoning, we propose a simple revision to the numerical process coupling scheme, which moves the application of the surface emissions to after dry removal and before turbulent mixing. The revised scheme allows newly emitted particles to be transported aloft by turbulence before being removed from the atmosphere, and hence better resembles the dust life cycle in the real world. Sensitivity experiments are conducted and analyzed to evaluate the impact of the revised coupling on the simulated aerosol climatology in EAMv1